The American Statistical Association (ASA) has finally spoken up about statistical significance testing.[1] Sort of.
Back in February of this year, I wrote about the simmering controversy over statistical significance at the ASA.[2] Back in 2016, the ASA issued its guidance paper on p-values and statistical significance, which sought to correct misinterpretations and misrepresentations of “statistical significance.”[3] Lawsuit industry lawyers seized upon the ASA statement to proclaim a new freedom from having to exclude random error.[4] To obtain their ends, however, the plaintiffs’ bar had to distort the ASA guidance in statistically significant ways.
To add to the confusion, in 2019, the ASA Executive Director published an editorial that called for an end to statistical significance testing.[5] Because the editorial lacked disclaimers about whether or not it represented official ASA positions, scientists, statisticians, and lawyers on all sides were fooled into thinking the ASA had gone whole hog.[6] Then ASA President Karen Kafadar stepped into the breach to explain that the Executive Director was not speaking for the ASA.[7]
In November 2019, members of the ASA board of directors (BOD) approved a motion to create a “Task Force on Statistical Significance and Replicability.”[8] Its charge was
“to develop thoughtful principles and practices that the ASA can endorse and share with scientists and journal editors. The task force will be appointed by the ASA President with advice and participation from the ASA BOD. The task force will report to the ASA BOD by November 2020.
The members of the Task Force identified in the motion were:
Linda Young (Nat’l Agricultural Statistics Service & Univ. of Florida; Co-Chair)
Xuming He (Univ. Michigan; Co-Chair)
Yoav Benjamini (Tel Aviv Univ.)
Dick De Veaux (Williams College; ASA Vice President)
Bradley Efron (Stanford Univ.)
Scott Evans (George Washington Univ.; ASA Publications Representative)
Mark Glickman (Harvard Univ.; ASA Section Representative)
Barry Graubard (Nat’l Cancer Instit.)
Xiao-Li Meng (Harvard Univ.)
Vijay Nair (Wells Fargo & Univ. Michigan)
Nancy Reid (Univ. Toronto)
Stephen Stigler (Univ. Chicago)
Stephen Vardeman (Iowa State Univ.)
Chris Wikle (Univ. Missouri)
Tommy Wright (U.S. Census Bureau)
Despite the inclusion of highly accomplished and distinguished statisticians on the Task Force, there were isolated demurrers. Geoff Cumming, for one, clucked:
“Why won’t statistical significance simply whither and die, taking p <. 05 and maybe even p-values with it? The ASA needs a Task Force on Statistical Inference and Open Science, not one that has its eye firmly in the rear view mirror, gazing back at .05 and significance and other such relics.”[9]
Despite the clucking, the Taskforce arrived at its recommendations, but curiously, its report did not find a home in an ASA publication. Instead, the “The ASA President’s Task Force Statement on Statistical Significance and Replicability” has now appeared as an “in press” publication at The Annals of Applied Statistics, where Karen Kafadar is the editor in chief.[10] The report is accompanied by an editorial by Kafadar.[11]
The Taskforce advanced five basic propositions, which may have been obscured by some of the recent glosses on the ASA 2016 p-value statement:
- “Capturing the uncertainty associated with statistical summaries is critical.”
- “Dealing with replicability and uncertainty lies at the heart of statistical science. Study results are replicable if they can be verified in further studies with new data.”
- “The theoretical basis of statistical science offers several general strategies for dealing with uncertainty.”
- “Thresholds are helpful when actions are required.”
- “P-values and significance tests, when properly applied and interpreted, increase the rigor of the conclusions drawn from data.”
All of this seems obvious and anodyne, but I suspect it will not silence the clucking.
[1] Deborah Mayo, “Alas! The ASA President’s Task Force Statement on Statistical Significance and Replicability,” Error Statistics (June 20, 2021).
[2] “Falsehood Flies – The ASA 2016 Statement on Statistical Significance” (Feb. 26, 2021).
[3] Ronald L. Wasserstein & Nicole A. Lazar, “The ASA’s Statement on p-Values: Context, Process, and Purpose,” 70 The Am. Statistician 129 (2016); see “The American Statistical Association’s Statement on and of Significance” (March 17, 2016).
[4] “The American Statistical Association Statement on Significance Testing Goes to Court – Part I” (Nov. 13, 2018); “The American Statistical Association Statement on Significance Testing Goes to Court – Part 2” (Mar. 7, 2019).
[5] “Has the American Statistical Association Gone Post-Modern?” (Mar. 24, 2019); “American Statistical Association – Consensus versus Personal Opinion” (Dec. 13, 2019). See also Deborah G. Mayo, “The 2019 ASA Guide to P-values and Statistical Significance: Don’t Say What You Don’t Mean,” Error Statistics Philosophy (June 17, 2019); B. Haig, “The ASA’s 2019 update on P-values and significance,” Error Statistics Philosophy (July 12, 2019); Brian Tarran, “THE S WORD … and what to do about it,” Significance (Aug. 2019); Donald Macnaughton, “Who Said What,” Significance 47 (Oct. 2019).
[6] Ronald L. Wasserstein, Allen L. Schirm, and Nicole A. Lazar, “Editorial: Moving to a World Beyond ‘p < 0.05’,” 73 Am. Statistician S1, S2 (2019).
[7] Karen Kafadar, “The Year in Review … And More to Come,” AmStat News 3 (Dec. 2019) (emphasis added); see Kafadar, “Statistics & Unintended Consequences,” AmStat News 3,4 (June 2019).
[8] Karen Kafadar, “Task Force on Statistical Significance and Replicability,” ASA Amstat Blog (Feb. 1, 2020).
[9] See, e.g., Geoff Cumming, “The ASA and p Values: Here We Go Again,” The New Statistics (Mar. 13, 2020).
[10] Yoav Benjamini, Richard D. DeVeaux, Bradly Efron, Scott Evans, Mark Glickman, Barry Braubard, Xuming He, Xiao Li Meng, Nancy Reid, Stephen M. Stigler, Stephen B. Vardeman, Christopher K. Wikle, Tommy Wright, Linda J. Young, and Karen Kafadar, “The ASA President’s Task Force Statement on Statistical Significance and Replicability,” 15 Annals of Applied Statistics 2021, available at https://www.e-publications.org/ims/submission/AOAS/user/submissionFile/51526?confirm=79a17040.
[11] Karen Kafadar, “Editorial: Statistical Significance, P-Values, and Replicability,” 15 Annals of Applied Statistics 2021, available at https://www.e-publications.org/ims/submission/AOAS/user/submissionFile/51525?confirm=3079934e.
